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Abstract 

An approach is proposed to the Hopfield model where the mean-field treatment is 
made for a given set of stored patterns (sample) and then the statistical average over 
samples is taken. This corresponds to the approach made by Thouless, Anderson and 
Palmer (TAP) to the infinite-range model of spin glasses. Taking into account the fact 
that in the Hopfield model there exist correlations between different elements of the 
interaction matrix, we obtain its TAP free energy explicitly, which consists of a series 
of terms exhibiting the cluster effect. Nature of the spin-glass transition in the model 
is also examined and compared with those given by the replica method as well as the 
cavity method. 
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1 Introduction 



Neural networks are systems in which a great number of neurons are connected with each 
other by synapses. The neurons basically take the two states, i.e., the firing and non- firing 
states. A neuron is firing if stimuli coming from (thousands of) neighboring neurons exceed 
a threshold. The neuron thus firing in turn affects neighboring neurons. These features are 
reminiscent of an Ising spin system with long ranged interactions. 

Hopfield [1] pointed out that the neural networks can be described by a mathematically 
equivalent model to that of spin glasses if couplings through synapses are symmetric and 
random. This suggests that the various methods developed for spin glasses are applicable 
to the neural networks. Indeed, numbers of studies have been made on this model since 
then [2-5]. Among others, the work made by Amit, Gutfreund and Sompolinsky (AGS) [5] 
is worth noticing. Applying the replica method, which is a mathematical trick to calculate 
the free energy, they investigated the Hopfield model to find that it exhibits a feature of 
the associative memory in a certain region in the T — a plane, where T is temperature and 
a = p/N is the ratio of the number of stored patterns p to that of neurons N. The region is 
called the retrieval feromagnetic (FM) phase. It was also shown by AGS that the model has 
another ordered phase, called the spin-glass (SG) phase, besides a disordered paramagnetic 
(PM) phase at highest temperatures. 

Sherrington and Kirkpatrick [6] proposed a model for spin glasses (the SK model) in 
which all Ising spins are coupled with each other through interactions which are given by in- 
dependent Gaussian random numbers. The model was introduced to construct the mean-field 
theory of spin glasses. Making use of the replica method, they obtained various properties of 
spin glasses. Although the original SK solution involves a difficulty to yield negative entropy 
at low temperatures, the model is now resolved by the replica-symmetry-breaking solution 
due to Parisi [7]. 

The replica method is successful, but it is rather abstract since, by this method, the aver- 
age over samples is carried out before examining thermodynamic properties of an individual 
sample. In order to get more direct physical insights of the SK model, Thouless, Anderson 
and Palmer (TAP) [8] developed the mean-field theory in the phase space, by which one first 
treats an individual sample and then takes the average over samples. They proposed the 
free-energy form which contains the effect of the 2-spin cluster besides the terms given by 
the conventional mean-field theory. The TAP free energy, properly derived afterwards [9,10], 
well works to further clarify various features of spin glasses such as the marginal stability of 
the SG phase [11], the many- valley structure in the free-energy landscape [12], the number 
of local free-energy minima [13] and so on. It is now known that the TAP free-energy ap- 
proach and the replica method are consistent with each other and provide complementary 
understandings of spin glasses [14,15]. 

The present work is motivated to develop such a TAP-like approach to the Hopfield 
model which is expected to play roles complementary to the AGS replica theory. Such an 
approach has been already described by Mezard, Parisi and Virasoro (MPV) in their text 
book [14] . Based on the cavity method, which they have successfully developed to derive the 
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TAP equations of states for the SK model, they have proposed the corresponding equations 
of states for the Hopfield model. We consider, however, that a part of their derivation has 
remained to be justified. 

The main purpose of the present paper is to derive the TAP free-energy expression for 
the Hopfield model directly by following the method due to Plefka [10], who derived the 
TAP free energy of the SK model. A crucial difference between the two models is that there 
exist correlations between different elements of the interaction matrix in the Hopfield model 
[14,16], while they are not found in the SK model. Consequently the TAP free energy of 
the former consists of an infinite series of terms exhibiting such correlation (cluster) effects. 
Based on the TAP free energy derived, we analyze mostly nature of the SG phase of the 
model and compare the results with those obtained by the replica method as well as by the 
cavity method. The derived TAP free energy is valid also in the retrieval FM phase, but the 
solution in this phase is left for a future study. 

In the next section we present the derivation of the TAP free energy of the Hopfield 
model. The PM-SG transition temperature T SG is calculated in section 3. Section 4 is 
devoted to some related discussions including comparisons of the present results with those 
obtained by AGS and MPV. 

2 Derivation of the TAP Free Energy 

Our starting Hamiltonian is 

H = -^2 JijSiSj, (1) 

where i{— 1, 2, • • • , N) denote spin (neuron) sites, and Si stand for spins (neurons) and take 
the values ±1; the value +1 and —1 correspond to the neuron which is firing and is not 
firing, respectively. The summation is taken over all spin (neuron) pairs. 
The interaction (synaptic efficiencies) J^ are given by 



Jij 



for i — j, 



where £f take ±1 and {£f } represent the /i-th stored pattern. Here we consider that £f are 
quenched, independent and random variables. This are also random variables. 

One sees that J^ obey the Gaussian distribution with = and J?- = p/N 2 , where the 
overline indicates the average over samples (different realizations of {Jij}, or (£f}'s). It 
should be noticed here that {Jij} are not independent with each other, but have correlations 
between different J^-'s [14,16]; for example, we see 

p a 



JijjjkJki — ^ 3 jy 2 - (3) 

These non-zero correlations bring about new terms in the free energy (see below). 
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In order to obtain the free energy, we follow Plefka [10]. Introducing external fields hf 1 , 
we consider 

H = aH-Y,h?Si. (4) 



Then, we make the Legendre transformation to get the free energy as a function of 



F = -TlnTr e ~ p6 + 



T m i- 



(5) 



Here T is the temperature (f3 = 1/T, with k-Q = 1) and m 8 = (Si) a , where (• • -) a denotes the 
expectation value with respect to H. We expand (5) with respect to a, i.e., 



n«) = E 



1 d n F 



n=0 



n\ da n 



a", 



a=0 



and then we put a — 1. Plefka showed 



dF 

da 
d 2 F 



and obtained 



dF 
da 

d 2 F 



da 2 



a=0 



a=0 



where we have introduced 



A, 



(H)a, 

-(3(H(H-(H) a -A 1 )) a , 



^ ' JijITliTTlj) 

-/3]Tjg(l-m 2 )(l-m 2 ), 



= E 



da" 
d (d n F s 



drrij \ da T 



(Si - mi). 



(6) 



(7) 
(8) 

(9) 
(10) 



(11) 



Now we extend the calculation up to the 4-th order. This calculation is fairly lengthy; we 
have made use of the algebraic programming system REDUCE-2. The results thus obtained 
are as follows: 



' = p(H)j^^ + P(HA 2 ) a + (3 2 (H(H - (H) a - A^, 



da 3 
d 4 F 



3/3 



da 
'd(H); 



+ P{H) a 



d 2 (H) a 



+ P(HA 3 ) a 



da J r N ' " da 2 
3(3 2 (HA 2 (H - (H) a - A 1 )) a - (3 3 (H(H - (H) a - A^a, 



(12) 



(13) 
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and 



{i,3) 

-6(3 2 ]T JijJjkJkiil-m^l- m^l -ml), (14) 

{i,j,k) 

-48/3 3 ^ J^J.Ml-m^il-m^il-ml) 

(i,j,k) 

x (JijUiirrij + Jjkmjrrik + J ki m k mi) 
-24(3 3 Yl JijJjkJktMl-m^l-m^l-ml^l-mj). (15) 

In the above, k) and (i, j, k, £) denote that the summation should be taken over inequiv- 
alent 3-spin clusters and 4-spin clusters, respectively. 

The free energy should be of the order of N, and therefore we have only to pick up 
terms proportional to N in (9), (10), (14) and (15). For the ferromagnetic Weiss model with 
Jij = 1/iV, we can see that only (9) gives the contribution proportional to N, as it should. 
In the SK model, the interactions { J^} obey the simple Gaussian distribution with = 
and Jfj = 0(1/N), and there is no correlation between different J^-'s. Therefore, as TAP 
pointed out, equations (9) and (10) give the contribution of the order of N. In the Hopfield 
model of present interest, equations (9) and (10) are of the order of iV as in the SK model. 
As mentioned in section 1, however, there exist correlations between different Ji/s. This 
provides new terms to the free energy. To show this, we take the last term of (14), as an 
example. Its order of magnitude is estimated as 

Y JijJjkJki{l-ml)(l-m])(l-m 2 k ) 

(i,j,k) 

N(N-l)(N-2) 

~ g <Jij<Jjk<Jki 

~ aN. (16) 

Similarly one can see that the last term of (15) yields the contribution of O(N). As for the 
other terms in (14) and (15), one can see that they can be neglected in the limit N — > oo. 
These analyses imply that d n F/da n \ a=0 for n > 5 also provide the terms of O(N), which are 
written in the form, 

- nW~ l " " " <W1 - <)(1 - <) •••(!- m 2 in ). (17) 

(h,i2,---,in) 

Their explicit derivation is given in Appendix A. 
As a result, we have the following free energy, 

F = Fq + -^duster, (18) 



d 3 F 



da 3 



a=0 



8 A F 



da* 



a=0 
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with 



2^ + T ( — — — 2 — — 2 — — 2 — ' ' ^ ' 



Cluster = -^£4(l-m2)(l-m2; 

Z (ij> 



oo 

ft Ji\ii^iii3 ' ' ' Jinh 

n=3 {«l,i2, ••■,««) 



(20) 

where the second term in (19) is the entropy, which comes from F(0) in (6). The TAP 
equations of states described in terms of {wij} are determined by dF/drrii = 0, i.e., 

T tanh -1 rrij — J^m^ — (3 ^2 ~ m f) m i 

3 3 

oo 

— 2 ^ /5 ^ j 2 • • • Jj n _ 1 % 

n = 3 (i\jl,j2,-,jn-l) 

x (1 - m J 2 J(l - ml) ■■■{!- mlJm,, (21) 

for % — 1, 2, • • • , N, where (i\ji,j2, • • • ,jn-i) means that the summation should be taken over 
inequivalent n-spin clusters with fixed i. With the substitution of m\ appearing explicitly in 
(21) by the spin-glass order parameter q = -/V -1 V^mf, equation (21) is rewritten as 

T tanh" 1 m 8 = V Jijirij - "^(^ g) m ( 2 2) 
j l-/3(l-q) 

In deriving (22) we have used 

(N — 1)(JV — 2) • • • (iV — n + 1) a 



'iji^im u 3n-n -- 2 iV n_1 

(iUlJ2,---,jn-l) 

= ^ (23) 
in which the factor 2 has been introduced, because we have 

^ih^hh ' ' ' Jjn-ii = Jijn-i ' ' ' ^hh Jjii- (24) 



3 Spin-Glass Transition Temperature 

Let us calculate the transition temperature, Tsg, which separates the normal (disordered) 
and spin-glass phases. To do so, we expand (22) up to the first order of m 8 and obtain 

Trrii = Jij m i ~ Y~[ mi - ( 25 ) 
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This implies that T SG is given by the equation, 

OL 

T SG + - J max = 0, (26) 

where J max is the maximum eigenvalue of the interaction matrix J. It should be mentioned 
here that the condition 

^max > 1 + 2v^ (27) 

should be satisfied to have real T S g- 

Our task is then to calculate J max - In Appendix B, it is shown that the distribution 
function of eigenvalues of J is given as follows: 

P (X) = \ KM + (1 " a)5(A + a) [° r a * ] (28) 
\ Po(A) for a > 1, v ' 

with 

m 1 y/(A- 1 + 2^(1 + 2^- A) 

Po(A) = — • -* — — , (29) 

27T A + a 

where A stands for eigenvalues of J. In Fig. 1 the behavior of po(A), a continuous part of 
p(A), is shown for some a. One notices at once that p(A) exhibits a quite different behavior 
from that of the independent Gaussian random matrix, for which it obeys the semi-circular 
law [11]. This is again the consequence of the non-zero correlations between the different 
matrix elements. For a < 1 p(A) consists of a delta peak at A = —a (whose amplitude is 
1 — a) and the continuous distribution po(A) around A = 1, i.e., 1 — 2^/a < A < 1 + 2y/a 
(whose integrated amplitude is a). Note that p(A) is normalized as / p(X)d\ = 1. At a — 1 
the delta peak merges to po(A), and for a > 1 p(A) exhibits a single and broad peak. As for 
the shape of po(A), we see from (29) that it becomes semi-circular and semi-elliptic for small 
and large a, respectively. 

In any a the largest eigenvalue is given by the upper edge of Po(A); J max = 1 + 2y / o~ 
Then we rewrite (26) to have 

(7 V-f >2 = 0. (30) 

This leads to T SG = 1 + -Ja. It is noted that J max thus obtained is just on the boundary 
of the condition (27), or T SG is given as a double root of (26). These circumstances are the 
same as those of the spin-glass transition temperature extracted by the TAP equation in the 
SK model. 

4 Discussion 

The spin-glass transition temperature T S g obtained by (30) coincides with the AGS result 
derived by the replica method. A further interesting comparison with the AGS result is on 
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the expression of the entropy. To show this, let us rewrite -F cluster of (20) in terms of the 
spin-glass order parameter q as we have done to derive (22). We obtain 

F cluster = ^aN{l-q + T ln[l - (3(1 - q)] }. (31) 
The entropy coming from F c i ustcr is then given by 

n f^cluster 



'cluster 



dT 



= A aN {^-^-rt + T^07)}- < 32 > 

This is exactly the same expression as that of the entropy in the limit T — > calculated by 
AGS (S = —(dF /dT) = in this limit). It becomes negative when it is evaluated in terms 
of the replica-symmetric solutions [5]. An expected proper solution is, as TAP argues for 
the SK model [8], that (1 — q) should vanish faster than T as T — > because we should have 
^cluster = at T = 0. 

In relation with the present result that T$g is determined as a double root of (26), let us 
consider the susceptibility matrix x%j — drrii/dhj. It is known that x is given by x — fiA~ x , 
where A is the Hessian matrix defined by A^ = d 2 ((3F) /dmidrrij. Then we see that Xmax 
diverges at T$g as Xmax — (T — Tsg)" 2 , where Xmax is the susceptibility of the eigenmode 
with the largest eigenvalue J max - The spin-glass susceptibility defined by xsg = X 2 
is calculated as 

XSG = J dX (T + a /(T X -l)-\f (33) 

in the PM phase. Since p(A) ~ (1 + 2y/a — A) 1 / 2 near its upper edge, we obtain xsg ~ 
(T — T S g) _1 - The replica method can provide the same result. These results described 
here indicate that nature of the PM-SG transition in the Hopfield model, including that the 
replica-symmetry-breaking takes place in the SG phase [5], is almost identical to that in the 
SK model. 

The TAP equations of state for the Hopfield model was already discussed by MP V [14] . 
They made use of the cavity method twice. In the first step, one spin is added to the A^-spin 
system, and the relations between quantities such as the free energy and the density of states 
of the N- and (N + l)-spin systems are examined to determine the distribution of field to 
the added spin. Then the following TAP equations are derived 



TOj = tanh p 



(34) 



where r 2 - n = N" 1 ££=i(faj) ~ (v^) 2 ) with ^ = N^ 2 For the SK model this 

step alone gives rise to the TAP equations of interest [14]. For the Hopfield model, on the 
other hand, MPV introduced another 'cavity method', in which the relevant relations are 
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those of quantities in the systems where p and (p + 1) patterns are stored. This yields, for 
the replica-symmetric solution, 



However equation (34) with (35) substituted does not coincide with our result, equation (22). 
Since the factor (3(1 — q) in the numerator of the second term of (22) is missing, the MPV 
equations do not reproduce the proper T$g- We suppose that the origin of the discrepancy 
would lie in the second step of the cavity method in the MPV argument. 

Finally we make a comment on the work by Geszti [2]. Starting from the equations 
rrii = tanh((3J2 Jij m j), he derived a set of the self-consistent equations for the retrieval FM 
order parameter m, the random overlap parameter r, and q, which coincides with those due 
to AGS derived by the replica theory. In his heuristic argument, however, the terms in (21) 
coming from -F c i ustcr of (20) are ignored. His argument is similar to the one by which the 
self-consistent equation for q of the SK model is derived, and which is criticized in [15]. A 
proper solution of (21) in the retrieval FM phase is our next concern. 

To conclude we have developed a TAP-like mean-field theory on the Hopfield model, by 
which we first analyze thermodynamics of individual sample with fixed { Jij}, or {£f }'s and 
then take the average over samples. In contrast to the SK model for spin glasses where only 
the 2-spin cluster effect is vital, it has been shown that a series of clusters, composing a large 
number of spins, play an important role in the Hopfield model. This gives rise to the TAP 
free energy which contains an infinite number of terms. Based on it we have investigated the 
PM-SG transition in the Hopfield model to find that its nature is almost identical to that in 
the SK model. We consider that the present TAP free-energy approach is useful in studying 
neural networks of a mean-field type since it will provide us complementary information to 
the replica method. 



9 



Appendix A. Derivation of equation (17) 



Here we discuss d n F/da n , from which we have terms of the order of N. We notice that the 
last terms of (14) and (15) come from the last terms of (12) and (13), respectively. Therefore 
we concentrate, in d n F/da n , on the term 

{-py-^HiH - (H) a - Ax)"- 1 ),. (Al) 

It is easily seen that d n F/da n contains the above term if one notices 



On the other hand, we have 



H - (H) a - A 1 = -J2 Jij(Si - mi)(Sj - rrij), 



(ij) 



n-l. 



and therefore (Al) can be written by 

- ( J2 J ij s i s j J2 J ij( s i - m i)(Sj - rrij) 
This provides equation (17) in the text, together with other irrelevant terms. 



(A2) 



(A3) 



(A4) 



Appendix B. Eigenvalue Distribution of J 

Following Bray and Moore [11], we write down the distribution function of eigenvalues of J 

as 



P(A) 



= — Im 

7T 



E G -( A - ie ) 



(Bl) 



where e is a positive infinitesimal and Gu are the diagonal elements of the matrix Green 
function 

G{\) = (A ■ i - J)-\ (B2) 
with 1 being the unit matrix. Then we make use of the so-called locator expansion to have 



1 1 



1 1 



X J A A 



1 . 1 



A A 



A 



= - + A + AA 2 + A 2 A 3 + 
A 



A(l-AA)' 



(B3) 
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Here A consits of an infinite series of terms due to the existence of correlations between 
different matrix elements of J (see section 2). Indeed, it is given by 

A = - (jijGJji) T + T (jijGJjkGJki) ~T 

3 A A (i\j,k) A 



Y {JijGJjkGJkiGJ^ — H 



A (»b\M) 

1 oo 



n=2 



(u|i2,23, ■■■,in) 



(B4) 



where we have introduced G by 



(B5) 



to take into account the renormalization. In the above, (ii\i2, 23, • • • , i n ) means that the 
summation should be taken over n-body cluster for fixed i±] we see 

J i\io.J i^iz ' ' ' Ji n i\ 

(u|i2,«3, ■■■,in) 

~ (N - 1)(N - 2) • • • (N - n + l)J ili2 J i2i3 ■ ■ ■ J inil 



~ a. 

Then we have 

From (B3), (B5) and (B7) we obtain 

G 

which is solved as 

1 



a G 



A 2 l-G' 



A[l - aG/X(l - G)] ' 



G 



A + l± v /(A + l) 2 -4(A + a) 



2(A + a) 

The above solution yields the imaginary part of G as follows: 



_ v/4(A + a - A + 1)2 

ImG = — ^ — ^ + 7cC5(X + a), 

2(A + a) 



where 



C 



1 — a for a < 1 
for a > 1. 



(B6) 



(B7) 



(B8) 



(B9) 



(BIO) 



(Bll) 



This result together with (Bl) and (B5) gives us equation (28) in the text. 
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Figure Captions 

Fig. 1: The distribution function po(\) for a =0.1, 0.5, 1, 1.5 and 2. 
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